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PREFACE 


The  research  documented  in  this  technical  report  was  sponsored  by  the  Air  Force 
Research  Laboratory,  Deployment  and  Sustainment  Division,  Logistics  Readiness 
Branch.  This  volume  is  the  first  of  three  volumes  that  summarize  work  performed  to 
develop  an  Aircraft  Battle  Damage  Assessment  and  Repair  (ABDAR)  technology  to 
enhance  the  capability  of  Air  Force  technicians  to  assess  damage,  determine  needed 
repairs  and  restore  the  aircraft  to  operational  status.  The  work  was  funded  under 
PE63106F,  Project  2745.  The  work  was  performed  under  contract  F41624-95-C-5003 
by  NCI  Information  Systems,  Inc.,  with  subcontractor  support  from  Boeing  Aircraft 
Company,  RJO  Enterprises,  Inc.,  and  GRACAR  Corporation.  Captain  Michael  Clark 
and  1st  Lieutenant  Steve  Grace  were  the  program  managers  for  the  major  portion  of  the 
effort.  Other  Laboratory  personnel  who  made  major  contributions  earlier  in  the  program 
were  Captain  Eric  Carlson,  Captain  Floyd  Gwartney,  1st  Lieutenant  J.C.  Bradford,  and 
1st  Lieutenant  Maurice  Azar. 

This  research  could  not  have  been  accomplished  without  the  support  and  assistance 
of  many  members  of  the  Combat  Logistics  Support  Squadrons,  the  Aircraft  Battle 
Damage  Repair  Program  Office,  and  the  Air  Force  Materiel  Command  Logistics 
Directorate  who  served  as  members  of  the  ABDAR  Users  Group,  provided  technical 
guidance  throughout  the  program,  and  provided  program  advocacy. 

The  653rd  Combat  Logistics  Support  Squadron,  Robins  AFB  provided  extraordinary 
support  for  the  program.  The  653rd  provided  the  test  facilities,  test  aircraft,  and  many  of 
the  technicians  who  participated  in  the  field  test.  The  squadron  also  provided  the 
support  of  several  of  their  instructors  who  served  as  subject  matter  experts  and  advisors 
throughout  the  program.  The  contributions  of  MSgt  Ken  McCain,  TSgt  Geoffrey  Miller, 
TSgt  George  Boutwell,  TSgt  Ken  Dockery,  and  TSgt  Rob  Meyers  as  technical  advisors 
were  invaluable  and  greatly  appreciated  by  the  ABDAR  program  staff. 

The  Executive  Summary  is  the  first  volume  of  a  three-volume  final  program  report.  It 
contains  a  summary  of  the  objectives,  assumptions,  methodology,  results,  conclusions, 
and  recommendations  of  the  Aircraft  Battle  Damage  Assessment  and  Repair  (ABDAR) 
program. 
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SUMMARY 

The  principal  objective  of  this  program  was  to  develop  and  evaluate  technology  to 
significantly  enhance  the  speed,  accuracy,  and  completeness  of  the  assessment  of 
battle  damaged  aircraft.  The  approach  adopted  was  to  develop  an  automated  capability 
to  provide  aircraft  battle  damage  assessors  with  technical  data  and  assessment  tools 
via  a  portable  maintenance  aid  (PMA).  A  demonstration  system  was  developed  and 
used  to  evaluate  the  Aircraft  Battle  Damage  Assessment  and  Repair  (ABDAR)  concept 
The  ABDAR  Demonstration  System  developed  for  the  field  test  was  an  end-to-end 
system.  It  started  with  the  aircraft  debrief  and  continued  through  the  ABDR  process  to 

/ACTo?CUmentati°n  °f  the  dama?e  assessment  on  an  Air  Force  Technical  Order 
•j  -9  ^orm  97-  The  system  design  was  based  upon  a  prioritized  set  of  requirements 
identified  by  the  ABDAR  Users  Group  (AUG).  The  demonstration  system  provided  the 
assessor  with  technical  data  for  the  testbed  aircraft  (F-15A),  including  applicable  F-15 
aircraft  battle  damage  repair  (ABDR)  manuals  and  Technical  Order  (TO)  1-1H-39, 
Technical  Manual  General  Aircraft  Battle  Damage  Repair.  The  system  supports  two 

LyP!Su?f  ^ectronic  Technical  Information  (ETI).  The  first  type  of  data,  in  the  Indexed 
Portable  Document  Format  (IPDF),  presents  technical  data  electronically  in  a  format 
very  similar  to  the  paper  TO.  The  second  type  of  ETI,  in  the  Content  Data  Model  (CDM) 
format,  provides  technical  data  in  an  interactive  mode.  Sample  technical  data  was 

developed  in  both  formats.  The  benefits  and  effectiveness  of  the  two  formats  were 
evaluated  in  a  field  test. 

A  field  test  was  conducted  at  Robins  AFB  to  evaluate  the  effectiveness  of  the 
system.  The  evaluation  was  accomplished  by  having  technicians  assess  simulated 
battle  damage  on  an  F-15  aircraft.  Two  thirds  of  the  technicians  assessed  the  damage 
using  the  demonstration  system,  and  one  third  of  the  technicians  performed  the 
assessment  while  using  the  paper  technical  orders.  Half  of  the  technicians  using 
electronic  technical  data  used  the  CDM  version,  and  half  used  the  IPDF  version.  Three 
types  of  technicians  performed  the  assessment  task.  They  were  fully  qualified  F-15 
battle  damage  assessors,  battle  damage  assessors  qualified  on  another  aircraft,  and 
technicians  (F-15  crew  chiefs)  who  were  not  trained  in  aircraft  battle  damage 
assessment.  Test  results  demonstrated  significant  benefits  to  using  the  ABDAR 
demonstration  system  for  both  the  IPDF  and  CDM  versions  of  the  technical  data. 
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AIRCRAFT  BATTLE  DAMAGE  ASSESSMENT  AND  REPAIR  (ABDAR) 

FINAL  PROGRAM  REPORT 
VOLUME  1:  EXECUTIVE  SUMMARY  REPORT 


INTRODUCTION 

The  ABDAR  final  program  report  consists  of  three  volumes. 

a.  Volume  1,  Executive  Summary,  contains  a  summary  of  the  objectives, 
methodology,  results,  conclusions,  and  recommendations  of  the  entire  program. 

b.  Volume  2,  Program  Methodology,  contains  an  overview  of  the  methodology  used 
to  satisfy  the  objectives  of  the  ABDAR  program. 

c.  Volume  3,  Field  Test,  contains  the  results,  conclusions,  and  recommendations, 
resulting  from  the  field  test. 

The  program  objective  was  to  develop  and  demonstrate  technology  that  would 
provide  a  significant  enhancement  in  the  capability  of  USAF  aircraft  battle  damage 
repair  (ABDR)  assessors  and  technicians  to  rapidly  assess  battle  damaged  aircraft. 
These  individuals  face  the  critical  task  of  assessing,  repairing,  and  returning  battle 
damaged  aircraft  to  mission  readiness  during  wartime.  The  ABDAR  program  built  upon 
the  concepts  and  technology  developed  in  the  Laboratory's  Integrated  Maintenance 
Information  System  (IMIS)  program  (Ward,  et  al.  1995,  Volumes  1,  2,  3,  and  Thomas, 
1995)  and  an  aircraft  battle  damage  assessment  concept  developed  in  an  earlier  study 
(Wilper,  et  al.  1983).  The  basic  approach  in  the  IMIS  program  was  to  provide 
technicians  with  a  portable  maintenance  aid  (PMA)  capable  of  presenting  all  technical 
and  diagnostic  information  required  to  perform  their  jobs.  A  similar  approach  was 
adopted  for  the  ABDAR  program.  Technology  and  a  demonstration  system  were 
developed  to  provide  the  ABDAR  assessor  with  information  and  planning  tools  needed 
to  perform  and  document  the  assessment  task. 

The  ABDAR  process  supported  by  the  demonstration  system  started  at  aircraft 
debrief  and  finished  with  final  documentation  of  the  damage  on  Air  Force  Technical 
Order  (AFTO)  Form  97.  The  ABDAR  demonstration  system  supported  the  Aircraft 
Battle  Damage  Repair  (ABDR)  process  by  providing  the  assessor  with  technical  data  for 
a  range  of  aircraft  systems.  The  data  was  derived  from  applicable  ABDR  manuals, 
including  TO  1-1H-39,  Technical  Manual  General  Aircraft  Battle  Damage  Repair.  The 
system  handles  two  types  of  electronic  technical  information  (ETI)  formats,  the  Indexed 
Portable  Document  Format  (IPDF)  and  the  Content  Data  Model  (CDM)  format. 

In  addition  to  development  of  the  ABDAR  demonstration  system,  the  ABDAR 
program  included  a  field  test  to  evaluate  the  effectiveness  of  the  system  and  determine 
its  benefits.  The  field  test  also  compared  the  benefits  of  the  two  types  of  electronic 
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technical  data.  To  accomplish  the  evaluation,  it  was  necessary  to  select  a  testbed 
aircraft,  inflict  simulated  battle  damage  to  the  aircraft,  and  develop  technical  data,  in  the 
electronic  formats,  needed  to  accomplish  the  assessment  task.  Fact  finding  visits  to  the 
Combat  Logistics  Support  Squadron  (CLSS)  units  and  other  information  led  to  the 
selection  of  the  F-15A  as  the  testbed  aircraft.  Damages  were  inflicted  on  an  F-15A 
aircraft.  The  damages  were  inflicted  at  door  R6  and  the  left  wing  trailing  edge.  The 
damages  involved  aircraft  systems,  structure,  and  wiring.  The  field  test  was  conducted 
at  Robins  Air  Force  Base  (AFB),  GA,  beginning  in  September  1998.  The  test  consisted 
of  two  distinct  phases:  Phase  I  -  ABDAR  process  Paper  Technical  Data  version  and 
Phase  II  -  ABDAR  process  ETI  version.  This  volume  of  the  report  describes  the  basic 
procedures  used  to  develop  the  ABDAR  demonstration  system  and  summarizes  the 
field  test  results. 


Purpose 

The  objective  of  the  ABDAR  program  was  to  design,  develop,  and  evaluate  an 
ABDAR  Demonstration  System  that  would  significantly  enhance  the  speed,  accuracy, 
and  completeness  of  the  assessment  of  battle  damaged  aircraft. 


Scope 

The  ABDAR  program  was  designed  to  allow  maximum  transfer  of  information  from 
one  phase  of  the  program  to  the  next  phase.  Significant  emphasis  was  placed  on 
involving  "real  world"  users  throughout  the  program  to  ensure  that  the  system 
developed  meets  their  needs  and  to  facilitate  product  transition  to  the  user  community 
The  overall  scope  of  the  ABDAR  program  included: 

a.  Identifying  and  defining  the  requirements  for  an  ABDAR  Demonstration  System. 

b.  Developing  specifications  that  define  the  ABDAR  Demonstration  System 
requirements. 

c.  Designing  and  developing  an  ABDAR  Demonstration  System  capable  of 
supporting  the  essential  ABDR  requirements. 

d.  Evaluating  the  ABDR  concept,  demonstration  system,  and  requirements  using 
the  ABDAR  Demonstration  System  in  a  field  test. 

e.  Incorporating  the  results  of  the  tests,  demonstrations,  and  evaluations  conducted 
into  the  final  report. 

ABDAR  PROGRAM  OVERVIEW  AND  APPROACH 

The  ABDAR  program  prime  contract  was  awarded  in  August  1995.  The  project  was 
conducted  in  three  phases;  requirements  analysis,  system  design  and  development, 
and  demonstration  system  implementation  and  field  test.  The  system  was  to  be 
capable  of  displaying  two  types  of  ETI  and  to  show  improvements  in  the  speed 


2 


accuracy,  and  completeness  of  damage  assessments,  over  the  current  ABDR  method. 
An  extensive  data  collection  effort  was  conducted  in  year  one  to  establish  user 
requirements.  System  design,  system  development,  and  the  field  test  were 
accomplished  in  years  two  through  four.  Planning  for  the  field  test  occurred  in  years 
two  and  three,  with  the  field  test  being  conducted  in  year  four,  from  September  1998  to 
October  1999.  Most  transition  planning  occurred  between  August  1998  and  December 
1 999.  The  overall  schedule  for  the  program  is  presented  in  Figure  1 . 


1995 

1996 

1997 

1998 

1999 

2000 

Phase  1  -  Requirements  Analysis 

Data  Collection 

▲ 

Requirements  Gathering 

A 

Technical  &  Literature  Review 

A 

Modeling 

A 

Phase  II  -  System  Design  and  Development 

Presentation  System  Review  &  Analysis 

A 

Program  Coordination 

A 

Authoring  System  Review  &  Analysis 

A 

User  Needs  Surveys 

A 

Design  Review 

A 

Phase  III  -  Demonstration  System 

Implementation  and  Field  Test 

Authoring  CDM  and  PDF  Data 

Field  Test  Planning 

“■ 

Pre-Field  Test 

Field  Test 

A  A 

Demonstrations 

A  warn 

Transition  Planning 

Figure  1.  High  Level  Schedule 

NCI  used  a  prototype  method  of  evolving  the  system  (referred  to  as  the  Interim 
Software  Demonstration  [ISD]  process).  Each  ISD  was  reviewed  at  the  ABDAR  Users 
Group  (AUG)  meetings,  where  user  feedback  was  obtained  and  plans  made  for  the  next 
iteration  of  software  development.  Figure  2  provides  a  pictorial  representation  of  the  ISD 
development  process  used  throughout  the  program. 
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ISD  Development  Process 


ABDAR  REQUIREMENTS  ANALYSIS 

A  systematic  process  was  used  to  identify,  record,  and  track  system  requirements 
during  the  initial  analysis  phase.  Requirements  definition  focused  on  the  end-user. 


Data  Collection 

Methods  used  to  gather  and  identify  requirements  included  interviews  with  ABDR 
personnel,  a  literature  review,  and  a  “user  needs  survey”. 

a.  interviews  with  ABDR  personnel  took  place  over  the  course  of  several  months. 
The  ABDAR  Team  visited  13  USAF  bases.  All  interviews  were  recorded  on 
audiotape,  providing  convenient  reference  in  areas  where  the  interviewer’s  notes 
were  not  clear.  After  each  data  collection  trip,  the  interviewers  compiled  their 

notes  and  submitted  them  to  a  data  modeler  for  entry  into  the  user  needs 
database. 


b'  S??  review  was  Performed  to  avoid  duplication  of  effort  with  other  existing 
.ABD;®™S'  Thls  review  was  also  designed  to  identify  other  potential  sources 
tor  ABDAR  Demonstration  System  requirements.  The  review  had  two  focal 
points;  reviewing  documentation  associated  with  IMIS  program  and  reviewing 
documentation  associated  with  aircraft  battle  damage  assessment  and/or  repair. 

c.  A  User  Needs  Survey  was  sent  to  11  active  and  reserve  Combat  Loqistics 
Support  Squadrons  (CLSS)  within  the  USAF.  A  total  of  133  surveys  were 
completed  by  experienced  ABDR  technicians.  Survey  items  were  listed  in  the 
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form  of  need  statements.  The  maintenance  personnel  assigned  a  rating  to  each 
item  using  a  scale  based  on  the  Modified  Cooper-Harper  scale,  traditionally  used 
to  measure  subjective  workload.  The  rating  was  based  on  their  level  of  need  for 
the  item  to  perform  their  assigned  tasks.  A  list  of  159  items  related  to  the 
capabilities  needed  to  accomplish  ABDR  tasks  were  extracted  from  the  data 
base  and  used  in  the  definition  of  the  ABDR  "As-ls"  process. 


Integrated  Computer  Aided  Manufacturing  (ICAM)  Definition  Methodology  (IDEF) 

The  IMIS  Architecture  (IMISA)  served  as  a  model  and  starting  point  for  development 
of  the  ABDAR  architecture.  Combining  the  IMISA  information  with  data  collected  in  the 
field  working  with  ABDR  subject  matter  experts  (SMEs)  resulted  in  a  realistic  portrayal 
of  the  ABDR  process  and  its  interfaces  to  the  organizational-level  maintenance 
environment  modeled  by  the  IMIS  program. 

Knowledge-Based  Systems,  Inc.’s  modeling  tools  were  used  to  model  the  functions, 
processes,  and  information  requirements  for  the  ABDR  domain.  Three  IDEF  models 
were  developed.  The  IDEFO  model  depicted  a  hierarchical  representation  of  the 
activities  and  information  flows  of  the  ABDR  world.  The  IDEF3  model  was  a  Process 
Flow  Network  (PFN).  The  ABDAR  IDEFIx  models  were  based  on  the  IMIS  “As-ls” 
IDEFIx  model  and  the  ABDAR  IDEFO  model.  Each  entity  and  relationship  in  the  IMIS 
“As-ls”  IDEFIx  model  was  examined  for  inclusion  in  the  ABDAR  IDEFIx  model. 
IDEFIx  was  useful  in  defining  database  structures  and  conceptual  models  during 
system  development. 


System/Subsystem  Specification  (SSS) 

The  ABDAR  SSS  defined  the  requirements  for  an  ABDAR  System.  The  ABDAR 
SSS  described  the  system  as  encompassing  activities,  processes,  and  information 
associated  with  technicians,  assessors,  team  chiefs,  and  maintenance  supervisors. 
The  SSS  defined  the  capabilities  required  for  an  operational  system. 


Identification  of  Demonstration  Requirements 

Using  the  ranking  structure  provided  by  the  ABDAR  User  Needs  Survey,  a  subset  of 
requirements  was  selected  from  the  ABDAR  SSS  for  use  in  developing  the  ABDAR 
Demonstration  System.  The  subset  of  requirements  was  selected  to  be  sufficient  to 
develop  a  demonstration  system  with  the  capabilities  and  features  required  to  effectively 
demonstrate  and  evaluate  the  ABDAR  concept.  In  October  1996,  the  recommended 
demonstration  system  requirements  were  presented  to  and  approved  by  the  AUG  and 
the  Government  Program  Manager  (PM). 

Another  key  aspect  of  demonstration  requirements  was  the  field  test  environment 
itself.  This  included  location,  type  of  aircraft,  types  of  damages,  aircraft  damage  sites, 
and  the  actual  damages.  Fact  finding  visits  and  other  information  obtained  resulted  in 
the  selection  of  Robins  AFB,  GA  as  the  test  site  and  the  F-15A  as  the  testbed  aircraft. 
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Damages  Involving  system,  structure,  and  wiring  were  inflicted  on  the  FI  5  at  locations 
door  6R  and  left  wing  trailing  edge.  These  sites  were  used  as  damage  sites  for  use  in 
the  evaluation.  These  decisions  determined  the  actual  data  requirements,  which  were  a 
significant  part  of  the  system  development  effort. 


ABDAR  DEMONSTRATION  SYSTEM  DEVELOPMENT 

The  ABDAR  Demonstration  System  was  developed  using  a  methodology  called 
evolutionary  prototyping.  Evolutionary  prototyping  is  a  life-cycle  model,  which  defines 
the  system  concept  as  the  product  is  developed.  Development  of  the  most  visible 
aspects  of  the  system  was  first.  These  visible  aspects  were  demonstrated  to  the  user 
through  an  Interim  Software  Demonstration  (ISD),  and  then  the  next  ISD  was  developed 
based  upon  user  feedback,  system  performance,  and  the  additional  requirements 
needed  for  a  fully  functioning  system.  The  evolutionary  prototyping  method  produced 
steady,  visible  signs  of  progress. 

The  development  team  produced  six  ABDAR  Demonstration  System  ISDs.  Four  of 
the  ISDs  were  used  in  developing  the  demonstration  system.  The  fifth  ISD  explored 
advanced  concepts  outside  the  requirements  of  the  field  test.  The  sixth  ISD 
documented  the  field  test  version  of  the  software.  Several  processes  were  utilized  in 
the  development  of  each  ISD.  These  processes  consisted  of  a  concept  phase,  code 
development,  data  development,  AUG  review,  human  factors  engineering  approach 
review,  and  an  analysis  phase. 


ISD  #1-  HyperText  Markup  Language  (HTML)  Prototype 

The  objective  of  ISD  #1  was  to  present  visual  objects  to  the  AUG  allowing  their  input 
to  guide  the  definition  of  the  functionality  and  appearance  of  the  ABDAR  Demonstration 
System.  The  assumption  was  made  that  by  incorporating  the  users'  perspective  early  in 
the  design  process,  the  ABDAR  team  would  be  better  able  to  ensure  that  the  final 
ABDAR  Demonstration  System  would  serve  the  users'  needs. 


ISD  #2  -  Damage  Collection 

AD.I^Drati0nale  for.ISD  #2  was  t0  be9in  system  development  on  the  portions  of  the 
ABDAR  Demonstration  System  that  were  considered  the  highest  risks  and  that 
contained  the  highest  potential  payoff  to  the  program.  The  most  significant  components 
that  concentrated  on  the  core  assessment  functions  were  Damage  Collection  and 
Repair  Planning.  Damage  Collection  was  the  primary  focus  of  ISD  #2. 


ISD  #3  -  Repair  Planning 

The  development  team  concentrated  on  developing  additional  essential  components 
in  ISD  #3  for  the  ABDAR  Demonstration  System.  Therefore,  Repair  Planning  was  the 
pnmary  focus  for  ISD  #3.  A  multi-tier  architecture  was  developed  for  use  in  ISD  #3  and 
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successive  ISD’s.  A  database  architecture  was  deployed  to  support  the  remaining 
iterative  prototypes  of  the  software.  The  database  was  implemented  to  evolve  along 
with  the  prototype  applications. 


ISD  #4  -  Field  Test  Version 

The  development  team  concentrated  on  producing  a  robust  version  of  the  system 
that  could  support  a  field  test.  The  biggest  risks  to  the  development  of  the  ABDAR 
Demonstration  System  were  an  immature  development  language  and  environment,  and 
the  ability  of  test  subjects  to  use  the  system  with  minimal  training.  To  mitigate  risks,  the 
ABDAR  Demonstration  System  was  modified  from  a  series  of  applets  to  a  single 
application. 


ISD  #5  -ABDR  Technology  Concepts 

Unlike  previous  ISDs,  ISD  #5  was  not  an  iterative  step  in  the  development  of  the 
ABDAR  Demonstration  System;  rather,  it  was  a  collection  of  thought-provoking  ideas  to 
enhance  the  ABDR  assessment  process.  The  ISD  #5  concept  and  analysis  paper 
documented  the  ideas  and  results  produced  during  that  effort.  The  ISD  #5  concept 
paper  is  provided  as  an  appendix  to  Volume  2  of  this  report. 


ISD  #6  -  Documenting  the  Field  Test  Version 

The  objective  of  ISD  #6  was  to  document  the  field  test  version  (ISD  #4)  of  the 
software  for  delivery  to  AFRL/HESR  at  program  end  as  ISD  #6.  No  major  software, 
hardware,  or  database  changes  were  required. 


FIELD  TEST  OVERVIEW 

The  field  test  provided  a  structured  approach  for  evaluation  of  the  ABDAR  concept 
and  Demonstration  System.  The  field  test  was  accomplished  at  Robins  AFB  between 
September  1998  and  October  1999.  The  specific  goals  of  the  field  test  were: 

a.  To  determine  if  the  ABDAR  Demonstration  System  improves  the  speed, 
accuracy,  and  completeness  of  the  ABDAR  process. 

b.  To  collect  data  that  clearly  identifies  benefits,  advantages  and  disadvantages  of 
the  ABDAR  Demonstration  System  for  supporting  the  damage  assessment 
process,  when  compared  to  the  current  paper  based  method. 

c.  To  evaluate  the  relative  performance  benefits  of  technical  data  developed  in  the 
IPDF  and  CDM  formats. 
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d.  To  identify  necessary  changes  in  the  ABDAR  SSS  to  provide  the  basis  for 
development  of  the  most  effective  battle  damage  assessment  system  for 
operational  implementation. 


Field  Test  Results 

The  field  test  compared  three  media  types  (CDM,  IPDF,  and  Paper).  Three 
categories  of  technicians  participated  in  the  evaluation.  They  were  ABDR  assessors 
qualified  on  the  F-15,  ABDAR  assessors  qualified  on  other  aircraft  (e.g.,  F-16,  C-130), 
and  F-15  Mechanics.  The  data  collected  was  analyzed  as  two  separate  experiments, 
with  data  from  the  ABDR  trained  subjects  evaluated  in  one  experiment  and  data  from 
the  F-15  mechanics  evaluated  in  the  second  experiment. 

The  first  experiment  was  a  2  x  3  factorial  design  with  technician  type  (F-15  Assessor 
and  Other  Assessor)  and  media  type  (Paper,  IPDF,  and  CDM)  as  the  independent 
variables.  Dependent  variables  were  time,  accuracy,  and  completeness.  Separate 
Analysis  of  Variance  (ANOVA)  tests  were  performed  on  each  dependent  variable. 
Subjects  for  the  first  study  consisted  of  30  USAF  maintenance  ABDAR  qualified 
technicians.  All  were  qualified  at  the  skill  level  seven  (7-level)  in  their  Air  Force 
Specialties,  Aircraft  General  Technician  and  Sheet  Metal  Specialist.  Fifteen  were 
ABDAR  qualified  F-15  assessors  and  15  were  ABDAR  qualified  assessors  on  other 
aircraft.  The  mean  scores  on  each  measure  are  summarized  in  Table  1. 

Table  1.  Summary  of  Scores  for  F-15  Assessors  and  Other  Aircraft  Assessors1 


Measure 

CDM 

IPDF 

Paper 

Time  (Minutes)2 

337.6 

629.6 

447.9 

Completeness  3 

95.4% 

90.3% 

56.8% 

Accuracy  4 

90.9% 

83.3% 

39.8% 

'  Scores  combined  for  F-15  and  other  aircraft 

assessors 

3  CDM  and  Paper  significantly  faster  than  IPDF  (p  <  .001) 

4  CDM  and  IPDF  significantly  higher  than  paper  (p  <  .001) 

CDM  and  IPDF  significantly  higher  than  paper  (p  <  .001 ) 

No  statistically  significant  differences  were  found  between  the  performances  of  the 
ADr%  anC*  °^er  a'rcr^ft  qualified  assessors  on  any  of  the  measures.  Subjects  using  the 
ABDAR  Demonstration  System  with  CDM  data  performed  significantly  faster  than 
subjects  using  the  ABDAR  Demonstration  System  with  IPDF  data,  improving  the  overall 
time  by  86%.  Assessments  produced  by  subjects  using  CDM  and  IPDF  data  were 
significantly  more  accurate  and  complete  than  those  produced  by  subjects  using  Paper, 
[®9frdless  of  Technician  Type.  Subjects  using  CDM  were  39%  more  complete  and 
51  /o  more  accurate  than  subjects  using  Paper.  Subjects  using  IPDF  were  34%  more 
complete  and  44%  more  accurate  than  subjects  using  Paper.  Overall,  the  ABDAR 
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Demonstration  System  tools,  in  conjunction  with  electronic  technical  data,  provided  a 
significant  advantage  over  the  current,  paper-based  method  of  performing  ABDR. 

The  second  experiment  was  a  1  x  3  design  with  F-15  Mechanics  performing  the 
same  tasks  as  above  with  the  three  media  types  (Paper,  I  PDF,  and  CDM).  Dependent 
variables  were  time,  accuracy,  and  completeness.  Separate  One-Way  Analysis  of 
Variance  (ANOVA)  tests  were  performed  on  each  dependent  variable.  Subjects 
consisted  of  18  USAF  maintenance  AFS  5  and  7-level  qualified  individuals.  The  mean 
scores  on  each  measure  are  summarized  in  Table  2. 


Table  2.  Summary  of  Scores  for  F-15  Mechanics 


Measure 

CDM 

IPDF 

Paper 

Time  (Minutes)1 

288 

580.3 

389.8 

Completeness  2 

95.0% 

92.8% 

55.2% 

Accuracy 3 

92.6% 

86.5% 

33.7% 

1  CDM  and  Paper  significantly  faster  than  IPDF  (p  <  .001) 

2  CDM  and  IPDF  significantly  higher  than  paper  (p  <  .001 ) 

3  CDM  and  IPDF  significantly  higher  than  paper  (p  <  .001) 


The  performance  times,  and  completeness  and  accuracy  scores  for  the  F-15 
Mechanics  were  very  similar  to  those  for  the  experienced  assessors.  Times  with  CDM 
and  paper  were  significantly  faster  than  with  IPDF,  with  no  significant  differences 
between  performance  times  for  CDM  and  paper.  Completion  and  accuracy  scores  for 
CDM  and  IPDF  were  significantly  higher  that  scores  for  paper.  There  were  no 
significant  differences  in  the  completeness  and  accuracy  scores  for  CDM  and  IPDF.  An 
informal  comparison  with  the  data  for  the  F-15  and  other  aircraft  suggests  that  there 
was  little  practical  difference  in  their  performance  and  the  performance  of  the  F-15 
mechanics.  The  implication  is  that  in  the  event  that  a  battle  damage  assessment  is 
required  and  a  qualified  F-15  or  other  aircraft  assessor  is  not  available,  an  F-15 
mechanic  can  effectively  perform  the  assessment  using  the  ABDAR  system.  This  could 
be  an  important  advantage  in  a  combat  situation  in  which  a  damaged  aircraft  is  forced 
to  land  at  a  base  with  no  qualified  assessors  available. 

Analysis  of  the  field  test  results  raised  the  question  of  whether  the  advantages 
observed  for  the  IPDF  data  were  due  to  the  tools/aids  provided  by  the  ABDAR  system 
or  simply  due  to  the  fact  that  they  were  presented  electronically.  Specifically,  were  the 
beneficial  results  found  for  the  IPDF  data  due  to  the  enhancements  (e.g.,  additional 
links)  made  to  the  data  and  the  aids  (e.g.,  wizards)  provided  by  the  ABDAR  system,  or 
were  they  due  to  presentation  of  the  information  electronically?  To  provide  some  data 
relevant  to  this  issue,  an  additional  data  collection  period  was  established  to  provide  a 
tentative  answer.  A  data  collection  team  returned  to  Robins  AFB  to  collect  data  from  a 
new  sample  of  F-15  Assessors.  Six  F-15  Assessors  performed  the  assessment  tasks 
under  the  same  test  and  evaluation  conditions  and  rules.  For  this  test,  subjects 
performed  a  conventional  assessment  with  the  use  of  Joint  Computer-aided  Acquisition 
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and  Logistics  Support  (JCALS)  standard  IPDF  TOs  instead  of  paper  TOs.  The  JCALS 
IPDF  data  was  presented  electronically  with  no  additional  enhancements  or  aids.  This 
setup  allowed  collection  of  data  that  would  be  comparable  to  the  ABDAR  enhanced 
IPDF  data  and  the  paper  data  collected  during  the  field  test. 

The  results  of  this  supplemental  data  collection  effort  are  summarized  in  Table  3. 
The  time,  completeness,  and  accuracy  data  were  consistent  with  data  collected  in  the 
main  part  of  the  study.  No  significant  differences  in  mean  times  were  found  for  any  of 
the  three  media.  The  mean  times  were  ABDAR  IPDF  -  594  minutes,  JCALS  IPDF  - 
486  minutes,  and  Paper  -  526.  For  the  completeness  measure,  subjects  using  IPDF 
scored  significantly  higher  than  both  subjects  using  JCALS  IPDF  and  subjects  using 
paper.  Subjects  using  JCALS  IPDF  scored  significantly  higher  than  subjects  using 
Paper.  ABDAR  IPDF  subjects  performed  significantly  better  on  the  accuracy  measure 
than  subjects  using  either  JCALS  IPDF  or  Paper.  There  were  no  significant  differences 
for  the  JCALS  IPDF  and  Paper  conditions. 


Table  3.  Comparison  of  Scores  for  ABDAR  IPDF,  JCALS  IPDF, 
and  Paper  Conditions  (F-15  Qualified  Assessors) 


Measure 

ABDAR  IPDF  JCALS  IPDF 

Paper 

Time  (Minutes)1 

594 

488 

526 

Completeness  2 

93.8% 

74.4% 

60.0% 

Accuracy  3 

88% 

58% 

44% 

No  significant  differences  in  times. 

ABDAR  IPDF  and  JCALS  IPDF  Completeness  scores  significantly 
Higher  than  scores  for  paper. 

3  ABDAR  IPDF  Accuracy  scores  significantly  higher  than  JCALS  and  Paper 
Accuracy  Scores,  no  significant  difference  between  JCALS  and  Paper  scores. 

As  would  be  expected,  the  JCALS  IPDF  data  demonstrates  an  improvement  in 
performance  times  over  paper  resulting  from  easier  access  to  data  and  elimination  of 
the  need  to  physically  search  for  and  acquire  the  data.  Improvements  are  also 
observed  for  the  Completeness  variable.  The  reasons  for  this  improvement  are  not 
obvious,  but  may  be  due  to  a  greater  reliance  on  the  use  of  technical  data,  since  it  was 
more  readily  available. 

After  completing  the  assessment  tasks,  using  the  ABDAR  Demonstration  system 
with  either  CDM  or  IPDF  data,  the  technicians  completed  a  short  questionnaire 
designed  to  obtain  their  reactions  to  the  system  and  to  solicit  any  suggestions  for 
improving  the  system.  The  responses  were  generally  quite  positive.  Technicians 
indicated  that  they  liked  using  the  system  and  believed  that  it  has  many  potential 
benefits  for  operational  use.  Several  expressed  a  desire  to  see  it  implemented  soon 
Some  typical  comments  are  given  below. 

Question  1.  What  aspects  of  the  ABDAR  System  were  most  helpful  to  you  in 
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performing  your  job?  Typical  responses: 

“I  like  how  it  takes  care  of  filling  out  the  AFTO  97's.  Really  like  the  way  it 
takes  you  to  the  specific  TO  area  you  need  to  be  in,  links  are  very  useful.” 

“The  speed  in  which  it  performs  the  task.  The  way  the  screen  won't  switch 
over  unless  you  process  the  work  correctly.” 

“K-factors  automatically  figured  for  you.  No  research  required  with  this 
system.  Extremely  quick  ordering.” 

“TO's  being  in  the  computer,  reduces  a  lot  of  time  looking  through  books.” 

Question  2.  What  aspects  of  the  ABDAR  System  did  you  dislike?  Typical  responses: 

“Not  having  detailed  parts  (information),  i.e.,  fittings  had  no  sizes  that  were 
recognizable  at  a  glance.” 

“Really  none,  just  takes  time  to  get  used  to  the  system.” 

“System  slowed  to  a  crawl  or  even  locked  up  after  a  few  hours  of  use.  I 
realize  that  the  system  is  still  under  design.” 

“When  a  qty  of  60  for  fasteners  is  needed,  then  when  you  allocate  you 
should  not  need  to  allocate  60  times  for  60  fasteners.  A  block  needs  to  be 
added  for  any  deviations  and  work  stoppage.” 

Question  3.  Are  there  any  changes  you  would  like  to  see  made  to  the  ABDAR  System? 
Typical  responses: 

“Sensor  touch  screens  in  which  you  can  use  a  pointer.” 

“Maybe,  install  a  track  ball  type  mouse.” 

“Better  user  friendly  controls  with  voice  and  video  links  (if  possible).” 

Question  4.  Are  there  any  other  concerns  or  comments  that  you  would  like  to  be 
known?  Typical  responses: 

“Make  one  system  and  improve  on  that  system.  TO’s  should  be  all 
standardized.  Not  fighters,  bomber,  airlift,  i.e.  If  27  is  for  flight  controls  then  it 
should  be  for  each  weapon  system.” 

“I  personally  need  more  training  to  give  a  better  viewpoint  on  what  we  really 
need.  Overall  it  is  a  vast  improvement  over  having  to  physically  search  for  info 
and  manually  document  forms.” 

“I  think  it  is  an  excellent  program.  I  think  it  should  be  helpful  to  the  assessors 
who  understand  the  setup  and  material.” 

“Make  it  so!  In  other  words  make  it  happen  as  soon  as  possible.” 
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DISCUSSION 

The  ABDAR  Demonstration  System  was  tested  in  a  simulated  ABDR  environment 
and  demonstrated  improvements  in  speed,  accuracy,  and  completeness  over  the 
current  ABDR  process.  Unequivocally,  assessors  using  either  CDM  or  IPDF  media 
types  far  outperform  assessors  using  Paper.  The  only  advantage  assessors  using 
Paper  have  is  a  slight  improvement  in  time  over  assessors  using  IPDF.  However,  due 
to  the  lack  of  completeness  and  accuracy  of  the  assessments  by  subjects  using  Paper, 
the  time  advantage  becomes  irrelevant.  In  other  words,  because  the  assessments  were 
not  as  complete  and  accurate,  it  made  little  difference  how  quickly  they  were  performed. 
It  is  likely  that  any  time  advantages  gained  through  paper  assessments  are  lost  when 
the  repair  process  is  initiated  and  the  repair  technicians  must  compensate  for 
incomplete  information  provided  by  the  assessment. 

Rapid  assessment  of  a  battle  damaged  aircraft  is  important,  however,  it  is  more 
important  that  the  assessments  be  complete  and  accurate.  Comparison  of  the  field  test 
data  for  the  different  types  of  media  leads  to  some  interesting  observations.  Figure  3 
presents  a  plot  of  the  Time  data  vs  Accuracy  Data  for  the  four  media  types.  The  figure 
illustrates  the  relationship  between  three  factors,  time,  completeness,  and  cost  (where 
CDM  has  the  highest  cost,  followed  by  ABDAR  IPDF,  JCALS  IPDF,  and  Paper).  The 
best  performances  in  terms  of  accuracy  are  provided  by  the  most  expensive  types  of 
data,  CDM  and  ABDAR  IPDF.  The  less  expensive  paper  technical  data  and  JCALS 
IPDF  data  were  much  less  effective  for  supporting  the  ABDR  assessment  task. 


Total  Assessment  Performance 
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Figure  3.  Plot  of  Accuracy  Scores  for  Four  Media  Types  for 
F-15  Qualified  Assessors 
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The  CDM  based  data  gives  the  best  overall  performance,  but  is  generally  more 
expensive.  For  new  weapon  systems,  which  will  already  have  their  technical  data  in  the 
CDM  format,  the  cost  of  adding  ABDAR  capabilities  will  not  be  great.  However,  to 
create  CDM  data  for  weapon  systems  (such  as  the  F-15)  would  be  cost  prohibitive, 
since  it  requires  completely  reformatting  and  restructuring  the  technical  data.  For  these 
weapon  systems,  the  development  of  ABDAR  I  PDF  would  be  the  most  cost  effective 
solution.  The  addition  of  performance  aids  and  enhancement  of  the  existing  JCALS 
IPDF  data  would  be  relatively  inexpensive  and  would  provide  major  improvements  in 
performance. 

The  performance  advantages  experienced  by  assessors  using  the  ABDAR 
Demonstration  System  with  the  CDM  and  IPDF  media  types  demonstrate  the 
advantages  that  would  be  realized  if  software  of  this  nature  were  implemented. 
Assessors  using  the  ABDAR  Demonstration  System  with  CDM  data  outperformed 
assessors  using  Paper  on  every  performance  measure  by  statistically  significant 
margins.  There  is  no  doubt  that  using  CDM  data  with  the  ABDAR  Demonstration 
System  is  a  radical  improvement  in  ABDR  methodology  compared  to  using  Paper. 
Assessors  using  the  IPDF  data  with  the  ABDAR  Demonstration  System  performed 
significantly  better  than  those  using  Paper  on  performance  measures  of  accuracy  and 
completeness,  faltering  only  on  time. 

Examination  of  Figure  2  clearly  suggests  that  the  use  of  CDM  data  is  preferable. 
The  development  of  CDM  data  for  new  aircraft  systems  is  not  significantly  more 
expensive  than  the  development  of  the  current  paper  or  JCALS  IPDF  data.  This  type  of 
data  is  being  developed  for  all  new  weapon  systems.  The  cost  of  adding  the  special 
data  elements  required  by  the  ABDAR  demonstration  system  to  this  data  should  not  be 
significantly  more,  a  relatively  small  investment  for  a  large  potential  gain.  The 
development  of  CDM  data  for  existing  systems  essentially  requires  reauthoring  the 
data,  which  can  be  quite  expensive.  The  use  of  IPDF  data  for  these  systems  is  a  viable 
alternative  that  gains  most  of  the  benefits  at  a  relatively  small  cost.  The  enhancement 
of  the  JCALS  IPDF  data  to  incorporate  the  ABDAR  requirements  is  relatively 
inexpensive,  considering  the  significant  benefits  in  completeness  and  accuracy  of  the 
assessments  produced. 

The  main  disadvantage  observed  in  the  field  test  for  the  ABDAR  IPDF  is  additional 
time  required  to  complete  the  assessments  (compared  to  the  CDM  and  paper  based 
assessments).  It  is  likely  that  this  problem  can  be  reduced  somewhat  by  some 
enhancements  in  the  user  interface.  Also,  it  was  observed  that  in  all  cases  subjects 
performed  faster  on  the  second  problem  that  they  performed  with  the  IPDF 
Demonstration  System.  This  suggests  that  some  of  the  performance  advantages  can 
be  overcome  with  additional  training  and  practice. 


13 


CONCLUSIONS 

The  results  of  the  ABDAR  program  development  and  field  test  support  the  followina 
conclusions: 

1 .  The  use  of  an  electronic  technical  information  system  with  the  capabilities  of 
the  ABDAR  Demonstration  System  significantly  enhances  the  ability  of 
technicians  to  perform  the  aircraft  battle  damage  assessment  task. 

2.  The  use  of  CDM  data  with  the  ABDAR  Demonstration  System  significantly 
improves  the  performance  of  the  assessment  task  by  all  categories  of 
assessors  on  all  measures. 

3.  The  use  of  I  PDF  data  with  the  ABDAR  Demonstration  System  significantly 
enhances  the  completeness  and  accuracy  of  assessments. 

RECOMMENDATIONS 

The  field  test  findings  support  the  following  recommendations: 

a.  An  electronic  technical  information  system  with  the  capabilities  demonstrated  in 
the  ABDAR  program  should  be  implemented  for  use  by  USAF  ABDR  personnel. 

b.  For  future  aircraft,  Air  Force  Materiels  Command  (AFMC)  should  ensure  that  the 
concepts  demonstrated  in  the  ABDAR  Demonstration  System  are  incorporated  in 
the  presentation  systems  currently  under  development.  All  future  weapon 
systems  should  utilize  CDM  data  from  the  beginning  of  the  program  and 
accommodate  the  ABDAR  System  needs. 

c.  Efforts  should  be  made  to  improve  the  speed  with  which  assessors  use  IPDF 
media  with  the  ABDAR  Demonstration  System.  Improving  the  IPDF  data  type, 
by  linking,  and  improving  the  IPDF  user-interface  should  reduce  the  time 
expended  performing  ABDR  with  IPDF.  Also,  improving  the  training  should 
improve  performance  times. 

d.  The  ABDAR  program  demonstrated  that  JCALS  data  can  be  effectively  used  to 
support  the  assessment  task.  Therefore,  AFMC  should  continue  its  efforts  to 
leverage  the  Joint  Computer-Aided  Logistics  Support  (JCALS)  initiative  to 
convert  paper  TO  data  into  electronic  format.  As  the  ETI  data  becomes  available 
from  JCALS  in  an  electronic  format,  AFMC  should  ensure  that  the  data  is  made 
available  to  all  the  ABDR  Combat  Logistics  Support  Squadrons  (CLSSs). 
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